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ABSTRACT 
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3. The Role of Amplified Objectives in Domain-Referenced 
Language Program Evaluation. 

Paper presented at the Regional Sem'nar of the SEAMEO Regional 
Language Centre in Singapore on Trends in Language Testing and 
Program Eva luati or. April 1990, and subsequently at the 
Post-RELC Regional Seminar organized by the Chulalongkorn Universitj 



The presentation dl vlded lnt0 ^ 

tr^ ctlon to aapimta objKtivte dMaln J •» 

-tin, and pro , r . M> fonowed hy ^ , au »«'• 

objectives, amplify objectlv „ ln t|)e context « «U« 

;;:;;::: on ' and - — - - — — - - 

INTRODUCTION 
Amplified Objectives 



9 

ERIC 



Amplified objectives are a significant attribute of the domain 
referenced model of program These ^ 

much finer end more specific th.n k k , " 
consist of , ^^ SPeCl " C than ^havioral ones. Amplified objective, 
consist of four dimensions: specifications of skills or sub-skill, 
specifications of content testi„™ . - , ' 
skins , w "chnlQues-dLcrete-polnt, macro- 

s i is. i„,i.. 8km test or integr.ted-sklll test, and specifications of 
-des of response - supply types, e.g.. short answers, fin . ln or 
selection types, e.g.. multiple choicei ^ 

amolifi? 1 ^' t0 S " lteila "-" t "^ test, a test developed fro. 
amplified ^actives will contain items measuring student performance on 
an objective or a set of related objectives with ♦ 
homogeneity. objectives with the property of 

Domain-refercnced Language Testing and Programs 

Domain-referenced testing is a system developed ^ ^ 

earning mastery which goes beyond criterion-referenced testing by not 
fo «ss „, only D „ th9 crU , rla of achuvement ^ ^ 

objectives, but also on the content/skill and the context of ovaiuati.n 
and the environment of learning and acquisition. A domain incorporates 
areas covered by the objectives, the content of learning, the skills or 
sub-skill levels (e.g.. speaking skill at the criticism level, reading 
skill at the comprehension level or the interpretative level,, student 
behavior and learning activities (<■ a . ... 

,„„., , ctivitles (e.g., taking notes, listening to 

lectures, and the media for learning (e.g.. textbook . audlo . tape> vlde0 . 
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tape, blackboard, etc.). The domain must have a definite area with clear 
borderlines separating it from other domains. In language testing and 
program evaluation, the domain is developed from the language syllabus by 
systematic sampling. m specifying a domain, it is necessary to include 
the following steps: 

(1) setting the limits of a domain 

(2) setting test types and test techniques as stimuli for 
student response 

(3) setting the modes and methods of response 

(4) setting the quantities of content/skills/sub-skills sampled 
from the domain of teaching and learning 

(5) setting the cut-off score of language competence 

A specified domain will set a clear limit to the field and range 
of objectives and skills/content of test items according to the steps 
spelled out above. The skills/contents of a domain-referenced test of 
listening skills may look like this: 

(a) listening to a conversation in a play 

(b) listening through the media of television 

(c) listening for pleasure/entertainment 

(d) listening connected with family affairs 

In developing a domain-referenced test the following parameters 
should be considered: 

(1) Context: time and location of language interaction, e.g.. in 
the classroom, at the train-station, in the restaurant etc. 

(2) Content Key: main idea or topic of language interaction, 
e.g., language in bargaining, in business transactions, in 
signing a treaty, in lecturing about biology, in advertising, 
etc. 

(3) Function and Discourse: language for work or personal 
purposes, elaborated/restricted code, referential/directive/ 
poetic/phatic function or metalanguage, 
descriptive/persuasive/explanatory discourse . 

(4) Modality, language interactions via the oral mode (listening/ 
speaking) or the visual mode (reading/writing/translation). 
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(5) Genre: methods of employing language skills, e.g. , Wrltin(j 
a personal letter, writing an essay, taking notes, writing 
a telegram, talking on the telephone, reading a newsp^ 
etc. • 

(6) Role: interactions between language transmitters an(J 
receivers, e.g., teacher-student, father-son, friend-friend, 
brother-older brother, etc. 

(7) Status: socio-economic statuses or professional areas 
governing the style and register of langauge use, e.g. 
ecclesiastical language, court language, vulgar language,' 
language for specific purposes-English for air-line 
hostesses, etc. 

(8) Pre-supposition: anticipation of language transmissions, 
e.g., expectancy of che incoming message through some media 
or a face-to-face interaction; familiarity with the language 
user or transmitter will facilitate communication, for 
example reading a friend's smeared letter with some words or 
phrases missing, or listening to one's mother in a noisy 
market place. Here communication is not hampered on account 
of the receiver's correct anticipation of the incoming 
message. 

(9) Mood and Attitude: the mood and attitudes of language users 
can determine the content and style of language used, e.g., 
humorous, rude, sincere, sarcastic, negative, positive, etc 

(10) Formality: the level of language used, e.g., formal, 
colloquial, slang, academic, standard, sub-standard, etc. 



The above parameters are embodied in the unit of meaning used to 
develop language test items to their full authenticity. The unit of 
meaning (B.J. Carroll, 1980) is composed of: 

(1) Situation: e.g., students attending a lecture on biology 

(2) Language Function: e.g., inquiry 

(3) Language Interaction: e.g., a student asking the 
teacher a question 

(4) Style: e.g., formal, polite 
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The unit of meaning: Inquiry (+ polite), (+ formal ) 
Language Manifestation: Excuse me, 1 wonder if you could 

explain about the nucleus of 

bacteria again. 

To conclude, we can say that the development of a domain- 
referenced language program as well as of a domain-referenced testing 
scheme for evaluating the psychological and sociological aspects of 
language interactions, provides the underpinnings for a language program 
aimed at communicative competence, by which student language performance 
is maximized through practice and use of language specified by the 
domain. (See examples in Appendices I and II developed from a set of 
learning objectives listed in Appendix III). 

THE DEVELOPMENT OF EDUCATIONAL OBJECTIVES 

This part will trace the development of educational objectives 
from the traditional model to the behavioral/performance model and the 
amplified model. 

Before 1920 the goals and objectives of education were abstract 
and elevated. There were no clear-cut terms to describe objectives. 
Imitation of the classical styles, appreciation of poetry and the 
upholding 0 f cultural heritage were the goals from which broad and hazy 
objectives were developed. 

With the advent of behaviorist psychology, in the 1950's language 
educators used the audio-lingual approach emphasizing the acquisition of 
"ew habits through practice. Precision of sound production and new 
v °cabuiary through mim-mem (mimicry-memorization) and gradual acquisition 
of reading and writing skills were at the center of the stage, 
drammed instruction developed from the rationale of Skinnerian 
Psychology which flourished in this era. 

Hazy and elevated objectives were denounced and replaced by the 

• measurable, precise and concrete behavioral objectives. They have 
Proved fr» k 

to oe very valuable for program development and evaluation. 

"Owever * 

. tor a domain-referenced language program in which social 



Interactions, attitudes, awareness of others' points of V iev 
anticipation of incoming message and the context of situations 
necessary parameters, behavioral objectives may not be very appropriate 
in the same way as Bloom's taxonomy of educational objectives, dividing 
them into three areas; cognitive, affective and psychomotor, will not be 
suitable for the evaluation of a language program which is cognitive 
skill-oriented. Kibler, Barker and Miles (1970, p. 75) stated that they 
believe that "taxonomies can be valuable educational tools but that they 
are not appropriate to all objectives, especially those involving complex 
behaviors." Baker (1974, p. 18) pointed out the drawbacks of behavioral 
objectives as incapable of being the bases for developing a syllabus, 'a 
good test and remedial teaching. This is because behavioral objectives 
lack the specification of representative content for syllabul/test 
development. A test will measure only whether the student has achieved 
the objective or not, while the test content may not at all be relevant 
to the syllabus or the goal of testing. Therefore, remedial instruction 
cannot stem from the results of a test measuring specific objectives, yet 
not having a well-defined domain of content/skill. To solve this 
problem, Baker recommended the use of amplified objectives used for 
domain-referenced testing. 

AMPLIFIED OBJECTIVES IN THE CONTEXT OF LANGUAGE 
PROGRAM EVALUATION 

Hively (1974, p. 6) emphasized the necessity of continual formative 
student evaluation to find out about student growth, strengths and 
weaknesses within the specified domain of skills and contents by usintj 
the results of tests developed from amplified objectives. 

As stated earlier, a program of language learning and testing 
cannot be specified within the domain of skills or contents alone; the 
sociological and psychological contexts of language interactions need to 
be included in th^ domain to be specified for language program 
development, language teaching and sampled for testing and evaluation. 
Carroll^ parameters ( 1980) incorporate Halliday «s three discourse 
dimensions (1978), i.e., field of discourse (register, style, topic, 
function, content of communication) mode of discourse (channel of 
communication, genre, modality) , and tenor of discourse (role 
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relationships among language users, socio-economic or social statuses, 
communication occasions governing language styles, e>9 ^ formal, frozen 
casual, and manners, e.g., polite, impolite). 

I 

, To illustrate the use of amplified objectives in language program 
evaluation, Stuf flebeam' s CIPP model is presented in the diagram below. 



Context 

riety 

ir.unity 

hool 

•her backgrounds 



Evaluation 
A.O 



Input 
Teacher/Variables 
Student Variables 
Materials 
Media, etc. 



Evaluation 
A.O 



Process 

Teaching/learning 
methods 

T — S, S—T, S— S 
S's~S ? s interaction 
Materials — S 
interactions 



Evaluation 
A.O 



Overall Evaluation 
Based on amplified objectives (A.O) 



Product 
Communicative 
Competence 



Evaluation 
A.O 



Amplified objectives (A.O) can be applied at every stage of 
Program evaluation. They form the basis for data collection, leading to 
the forming of judgments and the making of decisions. The properties of 
A «° which have maximal specificity enable the evaluator to detect the 
congruence, discrepancy, expected criterion and unexpected side~ef fects. 
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Context Evaluation of a Language Program 



At this stage, A.O specifying the contextual domain of langu age 
use, e.g., the business context, the academic context, the other soJo 
cultural dimensions of language interactions, are evaluated. Conte^ 
evaluation is open-ended in the sense tha'. its role is to provide the 
necessary information for program planning. 

Input Evaluation 

Variables integral to a program of learning include teacher 
variables, student variables, material/media variables, etc. A.O can be 
used to evaluate the 'nature, qualities, quantities, qualifications 
expectations, inclinations, study habits, etc. of human and non-huaan 
input variables. 

The A.O at this stage are planning objectives which are to be used 
with the A.O for context evaluation leading to setting of instructional 
A.O at the process evaluation stage. 

Process Evaluation 

At this stage instructional decisions are set to work by means of 
interactions between the teacher and the student, the student and the 
student(s), the teacher and the material/media, the student and the 
material/media. 

The A.O used to evaluate process expand the planning objectives 
into operational objectives focussing on actual implementation. Both the 
planned and the spontaneous or unplanned maneuvers are recorded and 
evaluated through employment of the A.O, This stage of prograa 
evaluation mirrors the teaching and learning operations against the 
backdrop of information obtained from context and input evaluations. 

Product Evaluation 

Student communicative competence manifested by his use of language 
within the domain specified by the A.O is evaluated by achievement tests. 

9 



Student learning mastery is evaluated against not only the criterion but 
the domain of expected cognitive skills and contents. 

The A.O used at this stage are outcome oriented. The outcome is 
evaluated against the criteria of performance. The criteria may be 
pass/fail or excellent, good, fair, poor, fail. Evaluation results are 
used for remediation and program revision. 

Product evaluation in the domain-referenced context will not be 
complete without the results of context, input and process evaluations. 
Therefore, an overall evaluation utilizing the combined sets of A.O at 
each stage should be conducted in order to obtain a completely valid set 
of evaluation results. 

The parameters. of planning, operational and performance A.O are 
identified as follows: 



Context & Input Evaluations <• 



Planning 
Amplified Objectives 



- Context & Input Specifications 



Phenomena 



Needs 



Expectations 



Concrete 



Experience 



Abstract 



Qualitatives- 



Quantitative data 



- Evaluation tools as stimuli 



Evaluation techniques 
questionnaire 
interview, etc. 
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- Type of responses 

oral report 
written report 
test score 
student profile 
teacher profile 
etc, 

- Criteria of planning evaluation 

methodological criteria 
(which techniques to bring 
about maximal results)? 
textual-media criteria 
(which course books, supple- 
mentary materials , visual 
aids f etc . will lead to 
maximal results)? 
evaluation criteria 
techniques and frequencies of 
formative evaluation) 

Operational 

Product Evaluation i Amplified Objectives 

- Specifications of interactions 

Types of interactions 
T-S 
3-S 

S-M (Materials) 
Channel of interactions 
English 
Thai 

oral/graphic 
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Study skills 
note-taking 
questioning 
responding 

getting meanings from 
various clues and cues 

expressing meanings 
etc. 

Task-based operational evaluation 

evaluation techniques 
types of test items 
frequency of evaluation 
content of evaluation 

Criteria of operational evaluation 

degree of congruence with 
planning amplified objectives 
(the greater degree of 
congruence, the better the 
operation) 



Types and frequencies of 
domain-referenced evaluation 



- weekly, monthly, mid-term 

- essay, multiple choice 

- discrete-point, global test 



Process Evaluation 



Outcome Oriented 
- Amplified Objectives 



Types of responses 



- supply 




- selection 




Criteria of performance 
pass-fail 

excellent very good, good, f aJr 

poor-fail 
(letter-grades, percentage) 



Overall Domain-referenced 4 sets of A.O. 

Program Evaluation 

Forming of judgement about the 
program 

Success 9 Failure 

Degree of success/failure 

Making decisions about thjB program 

- Remedial information 

- Recycling the input and the 
process with/without change 

- Discontinue the program 

- Institute a new program 



AMPLIFIED OBJECTIVES OF MATHAYOMSUKSA SIX 
(GRADE 12) ENGLISH PROGRAM 

Wiriya Sitthisarn, Sunanta Pakpian and Waraporn Sikhachai .1386) 
have conducted domain-referenced language program evaluations at the 
context stage In order to find out about the needs of students, teachers 
and parents for the English programs at Prathomsuksa (grade-school) or 
elementary level (grades 5-6), at lower secondary level (grades 7-9) and 
at upper secondary level (grades 10-12). Each developed approximately 
100 amplified objectives for each grade from the detailed analysis of the 
English curriculum for each level. For Mathayomsuksa Six, Wiriya 



constructs „3 „ pllfled objMtlves t0 u by ^ 

,„d teacher, on a s„ 3a 0 f 4 1„ order to indict, their needs for each 
it., in . doaain-r.ferenced English iancnag. progriB . Speciaens of ^ 

..plixied objectives for each sub-doa.in of coaaunicative competence are 
presented as follows: 



Needed Needed 
Least Most 



1. After listening to the economic, 
political, and sports news, the 
student is able to answer a four- 
choice multiple choice question 
about the main ideas and details 
of the news. 

2. After listening to the news, the 
student is able to make an oral 
summary of the news. 

3. While listening to descriptions 
of people, animals, objects and 
Places, the student is able to 
ask for more information about 
the objects of the description. 

«. While listening to narrations 

about incidents such as an accident, 
the student is able to take notes of 
what happened and able to sequence 
the incidents. 

f t« listening to songs, and poems, 
th * student is able to express his 
evaluative comments on what he has 
heard. 

When s °™eone makes a wrong 
statement about something, e.g., 
a Personal relationship, the 
Stud *nt is able to correct it. 
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7. The student is able to ask fov 
details about people, objects, 
places, time and activities, and 
is able to answer these types of 
questions. 

8. The student is able to make an 
appropriate oral invitation to 
various groups of people on 
various occasions. 

9. The student is able to speak 
about his determination to do 
something in an appropriate 
context or situation, e.g., when 
asked about his future plan for 
study. 

10. When a classmate speaks about 
something the student is not 
interested in, he is able to 
express his disinterest. 

11. After reading a letter or a note 
giving directions to a particular 
place, the student is able to 
draw a map of the place. 

12. After reading about a person's 
daily routine, the student is 
able to complete a time-table 
about that person's activities. 

13. After reading captions, the 
student is able to match the 
captions with pictures. 

14. After reading an assigned short 
story or novel, the student is 
able to relate it to others. 



Needed Needed 
Least Most 

12 3 4 



1 ? 3 4 



12 3 4 



12 3 4 



12 3 4 



12 3 4 



12 3 4 



12 3 4 
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Needed 



Needed 



t% After reading a congratulatory 
letter or card, the student is 
able to write a thank-you reply. 

16. After listening to a telephone 
message, the student is able to 
take the message down. 

17. After reading a short story, 
an article or a novel, the 
student is able to summarize it. 

18. After reading a short story or 
a novel, the student is able to 
extend it from his imagination. 

19. After reading a "WANT" ad, the 
student is able to write a 
letter of application. 

20. After receiving good/bad news 
about others, the student is 
able to write a letter of 

congratulation/sympathy to the 
person concerned. 



Least Most 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 

12 3 4 
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APPENDIX I 



Examples 



1. From the learning objectives of English Core r 

WU1 " of ' — - linguage progr ^TZIZZZT 6,5> 6,6 , ~* 121 — - 



Objective Language Mod e 
Function 



Genre 



Channel style 



Role 



Status Topic 



Ex, 1 



Summarize Listening Descripti 
& 

Speaking 



on Face to 
Face 



Formal Techer to Superi 
students 

as audience Inferior 



or People 



is ' 



0 
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3. 



Summarize Listening oia. 



Writing 



°* ue Tape casual Friend- 



recorder 



Equal 



Friend 



Invitation 
to a party 




u St0 „i„ g short play Radlo 



6l 

Writing 



Father- 
Son 



Superior Family 
quarrel 

Inferior 



APPENDIX II 



2. Fro. the learning objectives and the domain-referenced scheme 
p, progrem evaluation, prepare 20 outcome-oriented amplif led objeCtives 
specifying the communicative content, test types, and test techniques as 
stimuli, types and techniques of responses, and criteria. 

Example 



Learning 
Objective 



3. 



Amplified Objective 



After listening to statements made by the teacher about 
People in various professions, the student makes an oral 
summary of the statements in about 50-60 words. 

After listening to a taped dialogue between Bob and Jane in 
which Bob is inviting Jane to his party, the student 
summarizes the main points by writing 20-30 words. 



After listening to a short radio play about an argument 
between a son and a father, the student summarizes the play 
by writing 50-60 words. 



ID 



APPENDIX III 



Learning Objectives 
Core Course Eng. 615, 616 

After taking these courses, the student is expected to be able f 
perform the following: 



3. 
4. 
5. 
6. 

7. 



8. 

9. 

10. 
11. 



12. 
13. 
14. 

15. 
16. 

17. 
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List. , to statements with appropriate difficulty level and summary 
the main points. 

Listen tc dialogues with appropriate difficulty level and summarise 
the main points 

Listen to short pi ays and summarize the main points. 
Converse about daily activities. 
Converse about interesting events. 

Ask and answer by giving opinions about the statements and stories 
the student has listened to. 

Narrate personal experiences and stories the student has read by 
using vocabulary, expressions and grammar with appropriate level of 
difficulty. 

Enunciate words, phrases, sentences and statements using correct 
stress, intonation and rhythm to communicate intended meanings. 
Read statemen, , and assigned short stories with comprehension an* 
be able to express opinions. 

Read poems and summarize or express relevant opinions. 
Get the main points of, and express ideas or opinions about 
advertisements, announcements, notices, labels and instructions 
which the student has read. 

Read newspapers or magazines and discuss with friends. 
Take dictations. 

Write sentences using complex structures with appropriate vocabulary 
and expressions. 

Write statements and essays from guidelines or pictures given. 
Complete a conversation using pictures or statements about specific 
events. 

Write personal letters, business letters and fill out varl ous forn* 
by using appropriate expressions and punctuations. 
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1 g # write statements, stories or advertisements using the student's own 
ideas. 

19. Use English-English dictionaries appropriate to the student's level 
of English learning. 

20. Organize or participate in the cultural or daily activities of 
native speakers of English. 
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